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ABSTRACT 


This  paper  describes  a vision-based  method  for  avoiding  obstacles  using  the  concepts  of 
visual  looming  and  fixating  motion.  Visual  looming  refers  to  the  expansion  of  images  of  objects 
in  the  retina.  Usually,  this  is  due  to  the  decreasing  distance  between  the  observer  and  the  object. 
An  increasing  looming  value  signifies  an  increasing  threat  of  collision  with  the  object.  The  visual 
task  of  avoiding  collision  can  be  further  simplified  by  purposive  control  of  visual  fixation  at  the 
objects  in  front  of  the  moving  camera.  Using  these  two  basic  concepts  real  time  obstacle 
avoidance  in  a tight  perception-action  loop  is  implemented.  3D  space  in  front  of  the  camera  is 
divided  into  zones  with  various  degrees  of  looming-based  threat  of  collision.  For  each  obstacle 
seen  by  a fixating  camera,  looming  and  its  time  derivative  are  calculated  directly  from  the  2D 
image.  Depending  on  the  threat  posed  by  an  obstacle,  a course  change  is  dictated.  This  looming 
based  approach  is  simple,  independent  of  the  size  of  the  3D  object  and  its  range  and  involves 
simple  quantitative  measurements.  Results  pertinent  to  a camera  on  a robot  arm  navigating 
between  obstacles  are  presented. 

Index  Terms  - Looming,  Fixation,  Visual  Threat,  Obstacle  Avoidance. 
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1.  INTRODUCTION 


How  do  we  navigate  in  a world  full  of  stationary  and  moving  obstacles?  How  do  we 
avoid  collisions  with  furniture  and  moving  people  when  we  walk  from  one  end  of  the  room  to 
the  other  ? How  does  a driver  manipulate  his  speed  and  the  course  of  direction  during  driving 
a car  in  a heavy  traffic  ? 

We  argue  that  the  visual  looming  effect,  namely  the  expansion  of  objects’  size  in  the 
retina  is  essential  to  all  the  above.  Usually,  an  object’s  size  as  projected  on  the  retina  expands 
if  the  distance  between  the  observer  and  the  object  decreases.  This  visual  looming  carries  an 
indication  of  a possible  collision  with  that  object.  The  observer  may  react  defensively  to  reduce 
this  visual  threat.  Following  this  approach  it  is  possible  to  explore  navigation  problems,  in 
particular  the  obstacle  avoidance  problem,  for  an  autonomous  mobile  robot.  For  navigation, 
reconstruction  of  3D  world  is  computationally  expensive  and  may  not  be  needed.  In  many 
situations  a task-dependent,  tight  2D  based  perception-action  loop  can  be  adopted  [1-4]. 

The  problem  of  obstacle  avoidance  by  autonomous  mobile  robots  has  received  various 
approaches  from  researchers  in  the  computer  vision  area.  Usually  the  task  is  to  drive  the  robot 
from  a starting  position  to  a destination  through  multiple  obstacles.  In  some  cases  an  on-board 
sensor  is  not  used  and  the  environment  is  assumed  to  be  known.  An  approach  of  global 
optimization  to  generate  a trajectory  as  proposed  by  Gilbert  [5],  Shin  [6],  and  Kim  and  Shin  [7] 
requires  intensive  computation.  Introduction  of  artificial  potential  fields  that  attract  the  robot  to 
the  goal  and  repulse  the  robot  from  obstacles  has  been  proposed  by  Khatib  [8],  Hogan  [9], 
Krogh  [10,11],  Newman  and  Hogan  [12]  and  Volpe  and  Khosla  [13].  However  it  is  difficult  to 
implement  this  method  in  the  conventionally  steered  mobile  robots.  Nelson  and  Aloimonos  [14] 
introduced  an  alternative,  in  which  the  flow  field  divergence  is  used  as  a qualitative  measurement 
for  obstacle  avoidance. 

In  this  paper,  we  use  the  visual  parameter  looming  in  servoing  a camera  attached  to  the 
end-effector  of  a six-degree-of-freedom  robot  through  multiple  spherical  obstacles.  It  is  shown 
that  looming,  a vital  visual  cue,  when  used  actively  by  continuously  changing  the  points  of 
attention,  can  make  it  possible  to  navigate  in  a fairly  complex  environment.  Our  approach  does 
not  assume  any  knowledge  on  the  velocity  of  the  camera  and  the  size  and  range  of  the  obstacle. 
Looming  is  a measurable  parameter  that  can  be  calculated  directly  from  a sequence  of  2D  images 
obtained  from  an  on-board  camera.  Hence  the  idea  is  simple  and  readily  implementable. 

First,  we  provide  a summary  of  the  previous  results  on  looming,  including  definition  and 
methods  of  measuring  it.  Visual  fixation  and  looming  as  a function  of  time  is  covered  next. 
Finally,  the  obstacle  avoidance  algorithm  and  its  implementation  are  discussed  and  relevant 
results  are  presented. 
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2.  LOOMING  : SUMMARY  OF  PREVIOUS  RESULTS 


Looming  deals  with  expansion  of  the  object’s  image  on  the  retina.  Usually  this  expansion 
is  caused  by  a decreasing  distance  over  some  period  of  time  between  the  observer  and  the  object. 
Looming  has  been  studied  mostly  qualitatively  by  the  psychologists  [15-19].  Recently  a 
quantitative  approach  to  looming  has  been  presented  by  Raviv[20,21].  The  looming  value  of  an 
object  is  proportional  to  the  relative  translational  velocity  between  the  observer  and  the  object 
divided  by  the  distance  itself.  This  measurable  variate  can  be  extracted  directly  from  a sequence 
of  2-D  images  using  optical  flow  or  relative  change  in  area  of  the  image  of  the  object.  The 
concept  of  looming  is  also  similar  to  the  so  called  Nelson  and  Aloimonos’  flow  field  divergence 
[14]. 

2. 1  Mathematical  Definition  of  Looming  : 

The  looming  value  L of  a very  small  3D  object  is  defined  as  the  negative  value  of  the 
time  derivative  of  the  relative  distance  between  the  observer  and  the  center  of  the  object  divided 
by  the  relative  distance[21]  : 


dR 


R 


The  negative  sign  is  used  to  signify  image  expansion  with  positive  looming.  The  unit  by  which 
L is  measured  is  [time'*]. 

2.2  Equal  Looming  Surfaces  : 

Are  there  points  in  3-D  space  that  result  in  the  same  looming  value  L for  any  motion  of 
the  camera?  It  has  been  shown  [20,21]  that  such  points  for  a particular  looming  value  lie  on  a 
sphere.  The  center  of  the  sphere  is  located  on  the  instantaneous  translation  vector  and  the 
observer  is  located  on  a point  on  this  sphere.  A larger  translational  vector  will  produce  a larger 
sphere.  In  our  approach  objects  lying  on  an  Equal  Looming  Sphere  are  considered  to  pose  the 
same  threat  since  they  share  the  same  looming  value. 

2.3  Positive.  Negative  and  Zero  Looming  Surfaces  : 

Some  of  these  equal  looming  surfaces  correspond  to  positive  values  of  looming,  some 
correspond  to  negative  values  of  looming  and  there  is  a plane  (i.e.,  a sphere  with  infinite 
diameter)  that  correspond  to  zero  value  of  looming.  Figure  1 shows  the  three  types  of  surfaces. 
The  plane  that  passes  through  the  pinhole  point  of  the  camera  and  is  perpendicular  to  the 
instantaneous  translational  vector  contains  points  that  produce  zero  looming.  Points  on  the 
hemisphere  in  front  of  the  moving  camera,  (i.e.,  in  front  of  the  zero  looming  plane)  produce 
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positive  values  of  looming  and  likewise  points  on  the  hemisphere  behind  the  moving  camera, 
(i.e.,  behind  the  zero  looming  plane)  produce  negative  looming  values.  Naturally,  objects  lying 
on  the  zero  or  negative  looming  surfaces  do  not  carry  any  threat  of  collision. 


2.4  Measurement  of  Looming  Using  Optical  Flow  : 

Using  the  well-known  optical  flow  constraint  equation  [22],  an  expression  of  looming  has 
been  derived  for  a general  six-degree-of-freedom  motion  of  a camera  [21]. 

Assume  a stationary  environment,  with  a spherical  coordinate  system  (R,0,0)  attached 
to  a moving  camera  (Figure  2).  Let  the  instantaneous  angular  velocity  vector  w be  represented 
as  (A,B,C)^  where  T denotes  the  transpose.  Let  s^  = sind,  Cg  = cosd,  s^  = sm(f>,  c^  = cos4>  and 
L,  and  It  are  partial  derivatives  of  the  brightness  I with  respect  to  6,  and  time  respectively. 
Then,  the  expression  for  looming  can  be  written  as  : 


^^<l>pepe~^<t>pepd^^e  ^^<i>^^<t>pep<t>^e  '^^(j>pep4i^e 


where  the  translation  vector  t forms  6 = and  0 = angles  in  the  R-d-4>  coordinates. 

Hence,  given  the  location  of  the  pixel  in  the  image,  the  instantaneous  rotational  direction 
of  motion,  the  direction  of  the  instantaneous  translational  vector  and  the  spatial  and  temporal 
intensity  changes,  the  corresponding  looming  value  of  the  corresponding  point  in  3D  can  be 
obtained. 

2.5  Measurement  of  Looming  from  Relative  Rate  of  Expansion  : 

The  relative  rate  of  expansion  of  a small  object  in  the  image  is  proportional  to  its 
looming  value.  This  concept,  as  discussed  in  detail  in  [21],  shows  that  the  looming  can  be 
calculated  from  the  projected  area  of  a 3D  object  as  : 

dR  dA 
j dt  __  dt 


where  A is  the  projected  area  of  a 3D  ball  on  a spherical  image  and  R is  the  range  of  the  ball 
from  the  camera.  This  idea  of  relative  change  of  the  projected  area  is  used  in  our  approach 
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concerning  obstacle  avoidance  strategy. 


3.  VISUAL  FIXATION  AND  LOOMING  - AN  OVERVIEW 


3.1  Fixation  : 


Visual  fixation  is  actively  manipulating  the  imaging  system  by  directing  the  attention  to 
some  specific  points.  These  points  in  3D  space  carry  a pertinent  relationship  with  the  task  being 
performed.  The  fixation  point  may  be  stationary  or  in  motion  with  the  imaging  system.  By 
changing  fixation  or  the  point  of  attention  an  autonomous  system  can  reduce  a vast  3D  space  into 
a small  working  domain  and  hence  by  reducing  the  computational  complexity  it  can  simplify  the 
visual  tasks  involved.  Fixation  obviates  a very  high  resolution  wide  field  of  view  imaging 
system.  Moreover,  fixation  and  a logarithmic  retina  simplifies  the  calculation  of  looming 
[20,21].  The  relevant  concept  is  motivated  by  the  well  known  process  in  which  alternate  saccadic 
eye  movement  and  fixation  are  witnessed  in  human  obstacle  avoidance  [23]. 

3.2  Looming  and  Changing  Fixation  as  a Function  of  Time  : 

It  has  already  been  mentioned  that  equal  looming  surfaces  are  spheres.  The  size  of  each 
sphere  is  proportional  to  the  instantaneous  translational  velocity.  As  a camera  approaches  a 
point,  the  equal  looming  spheres  containing  this  point  shrink,  producing  higher  values  of 
looming.  The  centers  of  the  spheres  lie  on  a line  along  the  instantaneous  velocity  vector  (not 
necessarily  on  the  optical  axis)  and  the  observer  is  located  on  a point  on  the  sphere  surface.  Also 
it  has  been  shown  that  the  spheres  are  independent  of  the  instantaneous  rotational  parameters 
[20,21].  Now,  Figure  3a  illustrates  the  situation  where  a camera  moves  through  multiple 
stationary  objects,  fixating  at  them  and  calculating  each  obstacle’s  looming  value. 

The  camera,  while  making  the  translational  motion  rotates  and  fixates  at  each  of  the  four 
objects  A,B,C  and  D and  the  corresponding  looming  values  are  calculated.  The  looming  values 
are  plotted  in  Figure  3b.  L^,  Lg,  Lc  and  represent  the  looming  values  for  the  objects  A,  B, 
C and  D respectively.  Note  that  the  looming  values  of  all  the  four  objects  are  the  same  at  time 
instant  tj  since  they  lie  on  an  equal  looming  sphere.  Also  note  that  at  this  instant  of  time  the 
derivative  dL/dt  indicates  each  curve’s  rate  of  change  which  signifies  how  close  an  object  is  with 
respect  to  the  translational  vector.  dL/dt  is  the  maximum  for  D which  is  directly  on  the  path 
of  the  camera.  The  graph  also  shows  zero  and  negative  looming  values  at  some  time  instants. 

4.  THE  OBSTACLE  AVOIDANCE  ALGORITHM 


4. 1 The  Task  : 


The  task  to  be  performed  is  the  following  ; 

A camera  moving  in  a constant  translational  velocity  has  to  reach  a visible  goal  in  an 
environment  filled  with  several  obstacles.  The  camera  reaches  the  goal  when  the  looming  value 
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of  the  object  exceeds  a certain  value.  The  camera  is  assumed  as  a point. 


4.2  The  Concept  : 

The  space  around  the  camera  is  divided  into  the  following  three  zones  depending  on  two 
threshold  looming  values: 

- Safety  Zone 

- Mild  Risk  Zone 

- High  Risk  Zone 

This  is  illustrated  in  Figure  4.  Note  that  these  zones  are  moving  with  the  camera  and  their 
physical  size  may  expand  or  shrink  depending  on  the  speed  of  the  camera.  While  moving,  the 
camera  pans  and  fixates  at  each  visible  obstacle.  The  looming  value  is  calculated  for  each 
obstacle  using  the  method  of  relative  rate  of  change  of  area  of  the  projected  image  (Section  2.5). 

The  idea  used  is  that  varying  values  of  looming  signify  varying  degrees  of  threat  of 
collision  and  higher  looming  values  indicate  higher  degrees.  The  looming  value  of  each  obstacle 
is  compared  with  the  two  threshold  values  and  a risk  assignment  is  performed.  Refer  to  Figure 
5.  Obstacle  A carries  a high  risk  of  collision.  Similarly,  obstacle  B and  C pose  mild  and  no  risk 
respectively. 

The  camera  needs  to  alter  its  course  to  avert  possible  collision  with  an  obstacle  only  if 
the  obstacle  lies  in  the  High  Risk  Zone.  While  implementing  this  alteration,  the  course  is  refined 
taking  into  consideration  the  obstacles  lying  in  the  mild  threat  zone  too.  After  changing  the 
course,  the  camera  again  looks  at  all  the  visible  obstacles  lying  in  the  ± 90°  from  the 
instantaneous  translational  vector  and  decides  the  degrees  of  threat  coming  from  the  obstacles. 
If  necessary,  it  changes  its  course  again.  If  two  objects  enter  the  High  Risk  Zone  at  the  same 
time  the  algorithm  takes  into  account  the  derivative  of  the  looming  value.  Figures  6,7  and  8 
illustrate  these  sequences.  The  camera  advances  towards  the  obstacle  A until  the  obstacle  enters 
the  High  Risk  Zone.  After  changing  the  course  to  avoid  collision  it  pans  and  finds  a new  path 
through  B and  C.  This  path  is  safe.  Hence  it  continues  in  this  direction.  The  algorithm  uses  the 
following  notations  : 

Lhigh  = High  threshold  value  of  the  looming. 

For  any  obstacle  if  the  looming  value  exceeds  this  value,  then  the  obstacle  lies  inside  the 
High  Risk  Zone  and  the  camera  should  alter  its  course  to  avoid  any  collision. 

Lmiid  = MiW  threshold  value  of  the  looming  < Lbigt). 

If  the  looming  value  for  any  obstacle  is  in  the  range  of  l^gh  and  then  that  obstacle 
is  in  the  Mild  Risk  Zone.  It  is  not  needed  to  alter  the  course  of  the  camera,  but  if  it  is  altering 
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anyway  because  of  some  high  risk  obstacle,  it  should  refine  its  new  course  so  that  this  obstacle 
posing  mild  threat  does  not  become  a high  risk  threat  after  the  camera  changes  its  course. 

a = safe  passing  angle. 

This  is  a clearance  angle  that  the  camera  should  maintain  whenever  it  passes  through  two 
obstacles  that  it  is  trying  to  avoid. 

4.3  The  Basic  Algorithm  : 

step  1.  Initial  Path  : A straight  line  that  connects  the  start  and  the  destination  point 

is  the  original  path.  The  line  along  the  instantaneous  translational  motion  vector  is  the  current 
path.  Initially  the  current  path  and  the  original  path  are  the  same.  Start  moving  along  the 
original  path. 

step  2.  Fixation  : Rotate  the  camera  through  ±90°  (relative  to  the  current  path) 

fixating  at  each  visible  object. 

step  3.  Calculation  of  Looming  : Using  the  ’relative  rate  of  expansion’  method 

calculate  the  looming  L and  dL/dt  of  each  obstacle. 

step  4.  Risk  Assignment  and  Course  Deviation  Decision  : Sort  the  looming  values 

in  descending  order  and  compare  the  values  with  L^g,,  and  If  the  highest  looming  value  is 
in  the  High  Risk  Zone  then  alter  the  course.  That  obstacle  represents  the  maximum  threat.  If 
there  are  multiple  obstacles  with  the  same  looming  value,  then  they  lie  on  the  equal  looming 
sphere.  Look  at  their  time  derivative  dL/dt.  We  consider  only  positive  dL/dt  since  in  this  case 
the  looming  value  will  increase  as  opposed  to  negative  value  of  dL/dt  where  the  looming  value 
decreases.  The  obstacle  with  the  highest  dL/dt  lies  closest  to  the  current  path  and  poses  the 
maximum  threat.  If  the  highest  value  of  the  looming  is  in  the  range  L„,iid  ^ L < I^g^  then  it 
is  not  necessary  to  change  the  course.  If  the  highest  value  is  L < L^^yj  then  the  path  is  naturally 
safe. 


Hence,  if  L < Lhigi,  then  go  to  step  7 

else 

continue. 

step  5.  Changing  of  Heading  Vector  : Altering  the  course  is  necessary  to  avoid 

collision  with  the  obstacles  inside  the  High  Risk  Zone.  While  altering  the  course  ignore  the 
obstacles  lying  in  the  safe  zone. 

Rotate  the  camera  and  fixate  at  the  obstacle  posing  maximum  threat  (i.e.  maximum 
looming).  Measure  this  fixating  angle  from  the  current  path.  Also  measure  the  fixating  angles 
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for  other  obstacles  in  the  high  and  mild  risk  zones.  Choose  a path  that  is  closest  to  the  current 
path  and  maintains  the  safe  passing  angle  a. 

step  6.  Translation  : Move  in  this  altered  path  for  a unit  time.  Then  connect  the 

end  of  the  translational  motion  vector  with  the  destination  by  a straight  line.  This  becomes  the 

current  path. 

step  7.  Checking  the  Destination  : If  the  tip  of  the  translation  vector  is  not  within 

a specified  looming  value  with  respect  to  the  destination  point,  then  go  to  step  2 else  the 
destination  has  been  reached,  so  stop. 

Note  : Stay  as  close  as  possible  to  the  current  path  to  avoid  excessive  change  in  the 
steering  angle. 


5.  IMPLEMENTATION 


5.1  The  flight  simulator  : 

The  implementation  has  been  achieved  on  a miniature,  six-degree-of-freedom  vision-based 
flight  simulator.  The  simulator  allows  autonomous  vision-based  navigation  in  a miniature 
environment.  This  simulator  has  been  obtained  by  modifying  an  IBM  Clean  Room  Gantry  Robot 
(Figure  9).  The  modification  is  such  that  the  robot’s  main  controller  is  not  being  used,  instead 
it  is  controlled  directly  by  an  80486  based  personal  computer  that  supplies  analog  signals  to  the 
six  control  loops  each  corresponding  to  a velocity  input.  This  new  control  configuration  of  the 
robot  allows  continuous  and  smooth  motion  of  the  different  axes. 

The  visual  input  to  the  computer  is  obtained  from  a miniature  camera  located  at  the  end- 
effector  of  the  robot.  Visual  data  from  this  miniature  camera  is  processed  in  the  PC-based  vision 
processor.  As  a result,  pitch,  yaw,  roll  and  speed  signals  are  generated  and  sent  to  control  the 
robot. 

5.2  Experimental  Setup  : 

There  are  several  white  plastic  balls  placed  in  a dark  black  background  as  shown  in 
Figure  10.  Currently  the  camera  uses  only  the  roll  and  thus  pans  only  on  the  horizontal  x-z 
plane.  The  fixating  angle  can  vary  from  90°  positive  to  90°  negative  relative  to  the  current  path. 
The  simulator  looks  at  the  sequence  of  images  captured  by  the  camera,  continuously  changes  the 
fixating  angle  of  it  and  moves  through  the  white  balls  by  avoiding  collisions. 

6.  RESULTS 


A threshold  has  been  used  to  convert  the  sequence  of  captured  gray  level  images  into 
binary  levels.  This  simplified  the  task  of  identifying  the  white  obstacles.  Figures  11a  and  11b 
show  the  position  of  the  simulator  at  two  time  instants.  The  robot  advanced  by  avoiding  the 
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obstacles.  Figure  12  depicts  the  obstacles  as  seen  by  the  camera.  As  the  robot  moves  towards 
an  obstacle  the  change  of  projected  area  increases  prompting  it  to  alter  the  course  when  the 
obstacle  enters  into  the  High  Risk  Zone. 

7,  CONCLUSIONS  AND  FUTURE  WORK 


We  have  presented  that  visual  looming  as  a cue,  together  with  active  fixation  of  the 
camera,  are  useful  tools  for  obstacle  avoidance.  The  results  substantiate  that  hypothesis.  Our 
next  phase  of  work  will  cover  the  following  areas  : 

- calculation  of  looming  in  more  unstructured  scenario 

- considering  the  finite  size  of  the  camera 

- incorporating  speed  change  capabilities  of  the  camera 

- using  optical  flow 

- considering  moving  obstacles 

In  the  actual  implementation  of  our  idea  we  have  maintained  a somewhat  structured 
environment  with  white  balls  in  dark  background,  for  ease  of  locating  the  obstacles  and 
measuring  the  looming  values.  In  real  world  scene  naturally  the  world  will  not  be  full  of  white 
3-D  balls.  However,  most  of  the  real  world  objects  and  the  backgrounds  have  textures  which 
can  be  used  to  calculate  the  looming  values.  In  that  case,  separating  obstacles  from  the 
background  and  calculating  the  looming  for  each  of  these  obstacles  become  two  very  crucial  and 
challenging  problems. 
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Figure  1:  Positive  and  Negative  Looming  Spheres  and  Zero  Looming  Plane 


Figure  2:  Camera  Coordinate  System 


Figure  3:  Looming  and  Fixation  as  a Function  of  Time 


Figure  4;  Different  Looming  Zones 


Figure  5;  Obstacles  in  Different  Looming  Zones 


Figure  6:  Obstacle  A to  be  Avoided 


Figure  7:  Changed  Translational  Vector 


Figure  8:  Changed  Heading  Vector 


Figure  9:  Flight  Simulator 


Figure  10:  Experimental  Setup 


Figure  12:  An  Image  Taken  from  the  Moving  Camera 


